Neural Nearest Neighbors Networks

Non-local methods exploiting the self-similarity of natural signals have been well studied, for example in image analysis and restoration. Existing approaches, however, rely on k-nearest neighbors (KNN) matching in a fixed feature space. The main hurdle in optimizing this feature space w.r.t. application performance is the non-differentiability of the KNN selection rule. To overcome this, we propose a continuous deterministic relaxation of KNN selection that maintains differentiability w.r.t. pairwise distances, but retains the original KNN as the limit of a temperature parameter approaching zero. To exploit our relaxation, we propose the neural nearest neighbors block (N3 block), a novel non-local processing layer that leverages the principle of self-similarity and can be used as building block in modern neural network architectures. We show its effectiveness for the set reasoning task of correspondence classification as well as for image restoration, including image denoising and single image super-resolution, where we outperform strong convolutional neural network (CNN) baselines and recent non-local models that rely on KNN selection in hand-chosen features spaces.Comment: to appear at NIPS*2018, code available at https://github.com/visinf/n3net

Stochastic Variational Inference with Gradient Linearization

Variational inference has experienced a recent surge in popularity owing to stochastic approaches, which have yielded practical tools for a wide range of model classes. A key benefit is that stochastic variational inference obviates the tedious process of deriving analytical expressions for closed-form variable updates. Instead, one simply needs to derive the gradient of the log-posterior, which is often much easier. Yet for certain model classes, the log-posterior itself is difficult to optimize using standard gradient techniques. One such example are random field models, where optimization based on gradient linearization has proven popular, since it speeds up convergence significantly and can avoid poor local optima. In this paper we propose stochastic variational inference with gradient linearization (SVIGL). It is similarly convenient as standard stochastic variational inference - all that is required is a local linearization of the energy gradient. Its benefit over stochastic variational inference with conventional gradient methods is a clear improvement in convergence speed, while yielding comparable or even better variational approximations in terms of KL divergence. We demonstrate the benefits of SVIGL in three applications: Optical flow estimation, Poisson-Gaussian denoising, and 3D surface reconstruction.Comment: To appear at CVPR 201

Measuring and Removing Realistic Image Noise

When capturing photographs with a digital camera, the resulting images are inherently affected by noise. Image denoising, i. e. the task of recovering the underlying clean image from a noisy observation, is fundamental to improve the perceptual quality, to help further visual reasoning, or to guide the optimization for more general image restoration tasks. Since image noise is a stochastic phenomenon arising from different sources, such as the randomness introduced through the photon arrival process or the electric circuits on the camera chip, recovering the exact noiseless image is in general not possible. The challenge of the image denoising problem now arises by imposing suitable assumptions on both the formation process of the noisy image as wells as on the properties of clean images that we want to recover. These assumptions are either encoded explicitly within a mathematical framework that gives the denoised image as the solution of an optimization problem, or implicitly by choosing a discriminative model, e. g. a convolutional neural network (CNN), that is learned on training data comprised of pairs of clean and noisy images. Having defined a denoising algorithm, it is natural to ask for assessing the quality of the output. Here, the research community by and large relies on synthetic test data for quantitative evaluation where supposedly noiseless images are corrupted by simulated noise. However, evaluating on simulated data can only be a proxy to assessing the accuracy on realistic images. The first contribution of this dissertation fills this gap by proposing a novel methodology for creating realistic test data for image denoising. Specifically, we propose to capture pairs of real noisy and almost noiseless reference images. We show how to extract accurate ground truth from the reference image by taking the underlying image formation process into account. Since the image denoising problem is inherently ill-posed it is interesting to go beyond predicting a single possible outcome by additionally assessing the uncertainty of the prediction. Probabilistic approaches to image denoising naturally lend themselves for uncertainty prediction since they model the posterior distribution of denoised images given the noisy observation. However, inferring the quantities of interest, e. g. the marginal entropy at each pixel, is oftentimes not feasible. Our second contribution proposes a novel stochastic variational inference (SVI) algorithm that fits a variational approximation (Wainwright and Jordan, 2008) to estimate model-based uncertainty on the pixel level. We demonstrate that the resulting algorithm SVIGL is on par or even outperforms the strong baseline of SVI with the popular Adam optimizer (Kingma and Ba, 2015) in terms of speed, robustness, and accuracy. In this thesis we are also concerned with advancing the state of the art in terms of raw denoising accuracy. Currently, neural network based approaches yield the most powerful denoisers. Looking at more traditional methods, non-local approaches (Dabov et al., 2006) tend to be competitive. To combine the best of both worlds, in our third contribution we endow a strong CNN denoiser with a novel block matching layer, called neural nearest neighbors (N3) block, for which we propose a fully differentiable relaxation of the k-nearest neighbor (KNN) selection rule. This allows the network to optimize the feature space on which block matching is conducted. Our N3 block is applicable for general input domains as exemplified on the set reasoning task of correspondence classification. While the aforementioned parts of this dissertation deal with the common case of a saturating camera sensor, i. e. intensity values increase up to a maximal value, we also consider a novel sensor concept called modulo sensor (Zhao et al., 2015) that is promising for high dynamic range (HDR) imaging. Here, pixel elements reset once they reach their maximal value. To obtain a plausible image we need to infer how often each pixel was reset during the exposure. In our fourth contribution we particularly want to reconstruct this information from multiple noisy modulo images. We propose to faithfully model the image formation process and use this generative model in an energy minimization framework to obtain a reconstructed and denoised HDR image, outperforming prior approaches to reconstruction from multiple modulo images

Estimating Appearance Models for Multi-Person Tracking by Energy Minimization

In multi-target tracking, estimated tracks often erroneously fail to follow only a single target, hence causing identity switches. As a remedy, appearance cues are frequently exploited in order to make correct associations after a target has been temporally occluded or was not detected over several frames. In this thesis, we analyze multiple approaches of how appearance information can be integrated into the discrete-continuous energy minimization framework. To this end, we explicitly estimate appearance models for each trajectory and propose a generative as well as a discriminative energy to couple these appearance models with appearances of associated detections and trajectory frames. We show that both energies can be efficiently optimized with respect to the trajectory appearance model. Additionally, we analyze smoothness terms between appearances of consecutive trajectory frames and between labels of detections that exhibit a similar appearance. We conduct experiments under three aspects. First, we measure the general ability of the appearance energies to detect identity switches. Second, we qualitatively assess how the energy terms affect the overall tracking performance and third, we compare the full tracking system to state-of-the-art approaches on a variety of tracking scenarios. In order to lessen the influence of the optimization on the number of identity switches, we propose - and show its effectiveness - a novel recombination move that approximately minimizes the energy over the set of possible recombinations of short trajectory segments. Our analysis reveals that the benefit of the proposed energies mainly depends on how reliably appearance of a target can be measured. We see good results on scenes with clearly visible targets but observe a decline of performance for densely crowded scenes in which our system fails to estimate representative appearance models. Beim Verfolgen (Tracken) mehrerer Ziele passiert es oft, dass Trajektorien nicht ein einziges Ziel verfolgen und Identitätswechsel verursachen. Das Aussehen wird oft als Hinweis genutzt, um korrekte Zuordnungen machen zu können, nachdem ein Ziel kurzzeitig verdeckt war oder in einigen Bildern nicht detektiert wurde. In dieser Arbeit analysieren wir mehrere Ansätze, wie Informationen über das Aussehen bei der Minimierung einer diskret-kontinuierlichen Energie genutzt werden können. Für jede Trajektorie schätzen wir explizit Aussehensmodelle, die wir über eine generative bzw. diskriminative Energie mit dem Aussehen von zugeordneten Detektionen und Trajektorienbildern verknüpfen. Wir zeigen, dass beide Energien effizient in Bezug auf die Aussehensmodelle minimiert werden können. Eine weitere Energie bestraft ein sich änderndes Aussehen in aufeinanderfolgenden Bildern einer Trajektorie, während es eine vierte Formulierung bevorzugt, wenn Detektionen mit ähnlichem Aussehen demselben Ziel zugeordnet werden. Unsere Experimente untersuchen drei Gesichtspunkte. Erstens messen wir, wie gut mit den Energien Identitätswechsel erkannt werden können. Zweitens evaluieren wir die qualitativen Auswirkungen der Energien auf die Gesamtleistung des Tracking Systems, welches schließlich mit aktuellen Tracking Methoden verglichen wird. Um den Einfluss der Optimierung auf die Anzahl der Identitätswechsel zu reduzieren, schlagen wir einen neuen, effektiven Rekombinationsschritt vor, der annäherungsweise die Energie über der Menge der Kombinationen von kurzen Trajektoriensegmenten minimiert. Unsere Analyse zeigt, dass der Nutzen der vorgeschlagenen Energien wesentlich davon abhängt, wie zuverlässig das Aussehen eines Ziels beschrieben werden kann. Wir erzielen gute Resultate, wenn die Ziele gut sichtbar sind. Jedoch nimmt die Leistung des Tracking Systems in Szenarien mit vielen Menschen ab, da unser System dort nicht in der Lage ist, repräsentative Aussehensmodelle zu schätzen

Studies on Trueperella pyogenes isolated from an okapi (Okapia johnstoni) and a royal python (Python regius)

BACKGROUND: The present study was designed to characterize phenotypically and genotypically two Trueperella pyogenes strains isolated from an okapi (Okapia johnstoni) and a royal python (Python regius). CASE PRESENTATION: The species identity could be confirmed by phenotypic properties, by MALDI-TOF MS analysis and by detection of T. pyogenes chaperonin-encoding gene cpn60 with a previously developed loop-mediated isothermal amplification (LAMP) assay. Furthermore, sequencing of the 16S ribosomal RNA (rRNA) gene, the 16S-23S rDNA intergenic spacer region (ISR), the target genes rpoB encoding the β-subunit of bacterial RNA polymerase, tuf encoding elongation factor tu and plo encoding the putative virulence factor pyolysin allowed the identification of both T. pyogenes isolates at species level. CONCLUSIONS: Both strains could be clearly identified as T. pyogenes. The T. pyogenes strain isolated in high number from the vaginal discharge of an okapi seems to be of importance for the infectious process; the T. pyogenes strain from the royal python could be isolated from an apparently non-infectious process. However, both strains represent the first isolation of T. pyogenes from these animal species

Rationelle Energieverwendung

Die deutsche Energiepolitik wird weiterhin wesentlich von der Umsetzung der Energiewende und den sich daraus ergebenden Herausforderungen bestimmt. Durch die Wahlen zum 18. Deutschen Bundestag und die anschließende Bildung einer neuen Bundesregierung wurden Ende des Jahres 2013 die Weichen für die zukünftige Entwicklung gestellt. Verschiedene Maßnahmen und ein Bekenntnis zu mehr Energieeffizienz haben dabei auch Eingang in den Koalitionsvertrag gefunden. In zentralen Fragen wie der Umsetzung der Energieeffizienzrichtlinie oder der Höhe der Mittel für den nationalen Aktionsplan Energieeffizienz steht eine feste Positionierung der Bundesregierung allerdings noch aus. Auf europäischer Ebene ist aktuell das energie- und klimapolitische Zielsystem für das Jahr 2030 in der Diskussion. Ein eigenständiges Ziel für Energieeffizienz ist dabei im aktuellen Entwurf der Kommission nicht vorgesehen, das Ziel für Erneuerbare ist nur noch indikativ. Diese aus Sicht der Energieeffizienz bedenkliche Entwicklung kann sich aber im weiteren Verlauf der Gesetzgebung noch ändern

Automatic Detection of Song Changes in Music Mixes Using Stochastic Models

Plötz T, Fink GA, Husemann P, et al. Automatic Detection of Song Changes in Music Mixes Using Stochastic Models. In: International Association for Pattern Recognition, ed. Proc. ICPR 2006. Vol 3. Los Alamitos, Calif. : IEEE; 2006: 665-668.The annotation of song changes in music mixes created by DJs or radio stations for direct access in digital recordings is, usually, a very tedious work. In order to support this process we developed an automatic song change detection method which can be used for arbitrary music mixes. Stochastic models are applied to music data aiming at their segmentation with respect to automatically obtained abstract generic acoustic units. The local analysis of these stochastic music models provides hypotheses for song changes. Results of an experimental evaluation processing music mix data demonstrate the effectiveness of our method for supporting the annotation with respect to song changes